Examination assisting method, examination assisting apparatus, and computer-readable recording medium

ABSTRACT

A non-transitory computer-readable recording medium stores therein an examination assisting program that causes a computer to execute a process including: performing a site detecting process that uses an object detection technique on each of a plurality of ultrasound examination images taken of an examined subject by performing a scan on the examined subject; and displaying a site detection map in which a detection result of each of a plurality of sites included in the examined subject is kept in correspondence with the scan, on a basis of detection results from the site detecting process.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is based upon and claims the benefit of priority of theprior Japanese Patent Application No. 2018-157844, filed on Aug. 24,2018, the entire contents of which are incorporated herein by reference.

FIELD

The embodiments discussed herein are related to an examination assistingmethod, an examination assisting apparatus, and a computer-readablerecording medium.

BACKGROUND

Ultrasound examinations are known by which a subject is examined withoutbeing destructed as to whether the internal structure thereof has anabnormality or not. In an ultrasound examination, for example, atwo-dimensionally scanned cross-sectional plane of an examined subjectis imaged, so as to perform the examination by checking the image of thescanned cross-sectional plane. Because a probe used for the imagingprocess is operated by a person to perform the scan, for example, theimage of the scanned cross-sectional plane is strongly affected bychanges in the imaging environment. For this reason, in many situations,the image of the scanned cross-sectional plane, i.e., the ultrasoundexamination image is visually checked. Further, among techniques usedfor providing information useful for diagnosing processes, a techniqueis known by which a three-dimensional model is generated from a scanresult obtained by implementing Computed Tomography (CT), MagneticResonance Imaging (MRI), or the like, so as to present information aboutan arbitrary cross-sectional plane.

Further, object detection techniques are also known by which it isdetected what kind of object is rendered in an image. Among the objectdetection techniques, as methods for detecting an object in an image bymachine learning, for example, Deformable Parts Model (DPM) schemes andYou Only Look Once (YOLO) schemes have been proposed.

Non Patent Document 1: M. A. Sadeghi and D. Forsyth, “30 Hz ObjectDetection with DPM V5”, In Computer Vision-ECCV 2014, pages 65-79,Springer, 2014 Non-Patent Document 2: Joseph Redmon, Santosh Divvala,Ross Girshick, Ali Farhadi, “You Only Look Once: Unified, Real-TimeObject Detection”, arXiv:1506.02640v5 [cs.CV], 9 May 2016

However, ultrasound examinations involve checking each of a plurality ofsites from a plurality of images. Accordingly, there is a large burdenof performing an operation to select the images in which statuses of thesites are to be checked. In other words, the ultrasound examinationsimpose a large burden of judging whether an abnormality is present orabsent.

SUMMARY

According to an aspect of an embodiment, a non-transitorycomputer-readable recording medium stores therein an examinationassisting program that causes a computer to execute a process including:performing a site detecting process that uses an object detectiontechnique on each of a plurality of ultrasound examination images takenof an examined subject by performing a scan on the examined subject; anddisplaying a site detection map in which a detection result of each of aplurality of sites included in the examined subject is kept incorrespondence with the scan, on a basis of detection results from thesite detecting process.

The object and advantages of the invention will be realized and attainedby means of the elements and combinations particularly pointed out inthe claims.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory and arenot restrictive of the invention.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating an exemplary configuration of anexamination assisting apparatus according to a first embodiment;

FIG. 2 is a drawing illustrating an example of a site detection mapbased on an object detection technique;

FIG. 3 is a drawing illustrating an example of scanning methodsimplemented by a probe;

FIG. 4 is a drawing illustrating another example of the scanning methodsimplemented by the probe;

FIG. 5 is a drawing illustrating yet another example of the scanningmethods implemented by the probe;

FIG. 6 is a drawing illustrating yet another example of the scanningmethods implemented by the probe;

FIG. 7 is a drawing illustrating an example of a process of obtainingultrasound examination images;

FIG. 8 is a drawing illustrating an example of the site detection mapexhibiting a normal pattern;

FIG. 9 is a drawing illustrating an example of the site detection mapexhibiting an abnormal pattern;

FIG. 10 is a drawing illustrating another example of the site detectionmap exhibiting an abnormal pattern;

FIG. 11 is a drawing illustrating an example of the site detection mapexhibiting an undeterminable pattern;

FIG. 12 is a drawing illustrating another example of the site detectionmap exhibiting an undeterminable pattern;

FIG. 13 is a drawing illustrating an example of the site detection map;

FIG. 14 is a drawing illustrating examples of ultrasound examinationimages;

FIG. 15 is a drawing illustrating examples of site detection results;

FIG. 16 is a drawing illustrating another example of the site detectionmap;

FIG. 17 is a drawing illustrating other examples of the ultrasoundexamination images;

FIG. 18 is a drawing illustrating other examples of the site detectionresults;

FIG. 19 is a flowchart illustrating an example of an examinationassisting process according to the first embodiment;

FIG. 20 is a drawing illustrating examples of an examined subject and anexamination method according to a second embodiment;

FIG. 21 is a drawing illustrating examples of components that aresupposed to be rendered;

FIG. 22 is a drawing illustrating an example of a site detection mapaccording to the second embodiment; and

FIG. 23 is a drawing illustrating an example of a computer that executesan examination assisting program.

DESCRIPTION OF EMBODIMENTS

Preferred embodiments will be explained with reference to accompanyingdrawings. The present disclosure is not limited to the embodimentsdescribed below. Further, it is possible to combine together any of theembodiments described below, as long as no conflict occurs.

[a] First Embodiment

FIG. 1 is a block diagram illustrating an exemplary configuration of anexamination assisting apparatus according to a first embodiment. Anexamination assisting apparatus 100 illustrated in FIG. 1 performs asite detecting process that uses an object detection technique on eachof a plurality of ultrasound examination images taken of an examinedsubject by performing a scan thereon. On the basis of detection resultsfrom the site detecting process (hereinafter “detection results”), theexamination assisting apparatus 100 displays a site detection map inwhich a detection result of each of the plurality of sites included inthe examined subject is kept in correspondence with the scan. As aresult, the examination assisting apparatus 100 is able to provide thesite detection map that makes it possible to easily judge whether anabnormality is present or absent.

First, an object detecting process in the ultrasound examinationaccording to the present embodiment will be explained. In situationswhere ultrasound examinations are performed, objects serving as examinedsubjects have a certain internal structure in many situations. For thisreason, during the ultrasound examinations, it is possible to estimate astructure that is supposed to be present on the basis of informationabout positions of a probe or the like, by using knowledge fromexaminations, a design drawing, or the like. Accordingly, in the presentembodiment, with respect to an examined subject of which the internalstructure in a normal state is known, a site detection map indicating adetection result of each of the sites is generated by comparing aninternal structure detected by using an object detection techniqueemploying machine learning, with a normal structure that is supposed tobe present. In the current situation, although the object detectiontechnique using machine learning is not as good as the level of humanbeings (an average precision level mAP=approximately 80), it is possibleto calculate sufficiently reliable results by applying a statisticalprocess to processing results. Further, in the present embodiment, anexample will be explained in which a fetal heart will be used as anexamined subject. However, the present disclosure is also applicable toother organs and the like.

FIG. 2 is a drawing illustrating an example of the site detection mapbased on the object detection technique. As illustrated in FIG. 2, theexamination assisting apparatus 100 performs a site detecting process byapplying the object detection technique using machine learning, on aplurality of ultrasound examination images 11 taken of the examinedsubject by performing a scan thereon so as to obtain detection results12. As for the scanning direction, the scan may be performed in aforward direction, for example. However, the scan may be performed in areverse direction. On the basis of the detection results 12, theexamination assisting apparatus 100 generates and displays a sitedetection map 13 in which detection results of a plurality of sites(“sites A to F”) included in the examined subject are kept incorrespondence with “cross sectional planes 1 to 9” of the scan. In thissituation, the detection results are represented by probabilities of thesites. In the site detection map 13, each of the sites detected with asufficient probability is indicated with “S”, while each of the sitesdetected with an insufficient probability is indicated with “I”, andeach of the undetected sites is indicated with “U”.

Next, a configuration of the examination assisting apparatus 100 will beexplained. As illustrated in FIG. 1, the examination assisting apparatus100 includes a probe 110, a display unit 111, an operating unit 112, astorage unit 120, and a controlling unit 130. In addition to thefunctional units illustrated in FIG. 1, the examination assistingapparatus 100 may include any of various types of functional unitsincluded in known computers, e.g., functional units of various types ofinput devices and audio output devices.

The probe 110 is an example of a probing device that emits an ultrasoundwave toward the examined subject and receives an ultrasound wavereflected on the inside of the examined subject. As the probe 110, forexample, it is possible to use any of various types of probes such aslinear-type, convex-type, and sector-type probes. Further, for example,the probe 110 is capable of using an ultrasound wave with a frequency inthe range from 2 MHz to 20 MHz approximately. The probe 110 outputsreception data to the controlling unit 130.

Next, scanning methods implemented by the probe will be explained, withreference to FIGS. 3 to 6. FIG. 3 is a drawing illustrating an exampleof the scanning methods implemented by the probe. FIG. 3 illustrates anexample of a slide scan. The slide scan uses a scanning method by which,while being held vertically or horizontally, the probe 110 is movedparallel to a mother's body 20 so as to slide in an orthogonal directionwithout changing the angle thereof.

FIG. 4 is a drawing illustrating another example of the scanning methodsimplemented by the probe. FIG. 4 illustrates an example of a rotationscan. The rotation scan uses a scanning method by which, while using thecenter of the probe 110 as an axis, the probe 110 is rotated withoutchanging the rendering position with respect to the mother's body 20.

FIG. 5 is a drawing illustrating yet another example of the scanningmethods implemented by the probe. FIG. 5 illustrates an example of afan-shaped scan. The fan-shaped scan uses a scanning method by which,while the position of the probe 110 is maintained with respect to themother's body 20, a user swings the probe 110 in a fan-shaped region byusing his/her wrist.

FIG. 6 is a drawing illustrating yet another example of the scanningmethods implemented by the probe. FIG. 6 illustrates an example of apendulum scan. The pendulum scan uses a scanning method by which, whilethe probe 110 of a convex-type is being used, the probe 110 is swung inleft-and-right directions with respect to the mother's body 20, by usingthe semicircular curved surface in a convex shape.

Returning to the description of FIG. 1, the display unit 111 is adisplay device used for displaying various types of information. Forexample, the display unit 111 is realized as a display device by using aliquid crystal display monitor or the like. The display unit 111displays various types of screens such as a display screen input theretofrom the controlling unit 130.

The operating unit 112 is an input device that receives various types ofoperations from the user of the examination assisting apparatus 100. Forexample, the operating unit 112 is realized as an input device by usinga keyboard, a mouse, and/or the like. The operating unit 112 outputs anoperation input thereto by the user to the controlling unit 130, asoperation information. Alternatively, the operating unit 112 may berealized as an input device by using a touch panel or the like. Further,the display device of the display unit 111 and the input device of theoperating unit 112 may integrally be formed.

The storage unit 120 is realized by using a storage device, e.g., asemiconductor memory element such as a Random Access Memory (RAM) or aflash memory, or a hard disk, an optical disk, or the like. The storageunit 120 includes an image storage unit 121, an object data storage unit122, a learning model storage unit 123, and a site detection datastorage unit 124. Further, the storage unit 120 stores thereininformation used in processes performed by the controlling unit 130.

The image storage unit 121 stores therein a plurality of ultrasoundexamination images on the basis of the reception data input from theprobe 110. In this situation, the plurality of ultrasound examinationimages may be, for example, represented by a moving image having aplurality of frames. In the explanation below, the plurality ofultrasound examination images may be referred to as a “moving image”,while an image in a frame of the moving image may be referred to as an“examination image” or an “ultrasound examination image”.

The object data storage unit 122 stores therein object data indicatingthe structure of an object serving as an examined subject. As the objectdata, it is possible to use, for example, rule-based data estimated byusing the current position of the probe 110 and a relationship between apreceding frame and a following frame, or the like. Alternatively, asthe object data, it is also acceptable to use data based on athree-dimensional model or the like based on a manual input or a designdrawing. In other words, the object data may be expressed as a set R_tmade up of sites that are supposed to be rendered in an examinationimage m_t taken at a time t, the set R_t having been obtained by using acertain method. Accordingly, in the explanation below, the set R_t madeup of the sites that are supposed to be rendered in the examinationimage m_t taken at the time t may also be referred to as object dataR_t. Further, a time period of times t corresponding to examinationimages m_t of which the object data R_t is present is expressed as atime set T_R. A set made up of the examination images m_t correspondingto the time set T_R will be expressed as all examination images M. Inother words, the object data storage unit 122 stores therein the timeset T_R together with the object data R_t. The object data R_tcorresponding to the time set T_R will be expressed as object data R.

With regard to the object serving as the examined subject, the learningmodel storage unit 123 stores therein a learning model obtained bylearning a plurality of elements h related to a structure H of theobject. The learning model is obtained by learning, in advance, theelements h of the structure H of the object, by using an objectdetection algorithm such as YOLO, a Single Shot MultiBox Detector (SSD),a Faster-Recurrent Neural Network (RNN), or the like. For example, thelearning model stores therein various types of parameters (weightcoefficients) of a neural network, or the like.

With respect to each of the examination images m_t, the site detectiondata storage unit 124 stores therein site detection data, which is dataobtained by detecting the elements h of the structure H of the object byusing the learning model. It is possible to express the site detectiondata as a set D_t made up of sites (elements h) rendered in theexamination image m_t taken at the time t. Accordingly, in theexplanation below, the set D_t made up of the sites rendered in theexamination image m_t taken at the time t may also be expressed as sitedetection data D_t. Further, the site detection data D_t may also beexpressed as a probability map D(h,t), when a focus is placed on a setmade up of probabilities P_h(m_t) of the sites (the elements h) renderedin the examination image m_t taken at the time t. In other words, theprobability map D(h,t) is a set made up of probabilities P_h(m_t) of thesites (the elements h) rendered in the examination image m_t taken atthe time t. That is to say, the probability map D(h,t) is an example ofthe site detection map. The site detection data D_t corresponding to thetime set T_R will be expressed as site detection data D.

For example, the controlling unit 130 is realized as a result of aCentral Processing Unit (CPU), a Micro Processing Unit (MPU), or thelike executing a program stored in a storage device provided on theinside thereof, while using a RAM as a work area. Alternatively, thecontrolling unit 130 may be realized by using an integrated circuit,e.g., an Application Specific Integrated Circuit (ASIC), a FieldProgrammable Gate Array (FPGA), or the like. The controlling unit 130includes an obtaining unit 131, a judging unit 132, a detecting unit133, and a display controlling unit 134 and realizes or executesinformation processing functions or actions described below. Possibleinternal configurations of the controlling unit 130 are not limited tothe configuration illustrated in FIG. 1. The controlling unit 130 mayhave a different configuration as long as it is possible to implementthe information processing processes described below.

When being instructed to start obtaining a moving image, the obtainingunit 131 starts obtaining the reception data from the probe 110. On thebasis of the obtained reception data, the obtaining unit 131 startsgenerating the moving image. In other words, the obtaining unit 131starts obtaining the moving image. The examination images based on thereception data are obtained by calculating a distance by using a timeperiod between when an ultrasound wave is emitted and when a reflectedwave returns. For example, it is possible to use any of various types ofexamination images such as B-mode, M-mode, and color Doppler images.Further, when the obtained moving image has a section in which thescanning direction goes backward due to a movement of the fetus or thelike, for example, the obtaining unit 131 eliminates one or more framescorresponding to the backward part by comparing the frames of the movingimage with one another. The obtaining unit 131 stores the obtainedmoving image into the image storage unit 121 and also outputs theobtained moving image to the judging unit 132.

Next, the process of obtaining the moving image, i.e., the process ofobtaining the plurality of ultrasound examination images, will beexplained with reference to FIG. 7. FIG. 7 is a drawing illustrating anexample of the process of obtaining the ultrasound examination images.In the example in FIG. 7, the examination images (the ultrasoundexamination images) are obtained by scanning the abdomen of the mother'sbody 20 with the probe 110, with regard to a fetus 21 inside themother's body 20. For example, the probe 110 is operated by a medicaldoctor to continuously perform a scan starting from the vicinity of thestomach of the fetus 21 toward the upper section of the heart. As theexamination images in this situation, examination images 22 to 24 areobtained as examples of examination images before and after the heartand an example of an examination image of the heart. The examinationimage 22 renders ribs 25, the spine 26, the descending aorta 27, thestomach 28, and the umbilical vein 29.

The examination image 23 renders ribs 25, the spine 26, the descendingaorta 27, and the heart 30. Further, the examination image 23 alsorenders sites of the internal structure of the heart 30, namely, theright ventricle 31, the right atrium 32, the left ventricle 33, the leftatrium 34, the interventricular septum 35, and the crux cordis 36. Inthis situation, the heart 30 corresponds to the structure H of anobject, i.e., a set made up of structures of the heart. Further, theright ventricle 31, the right atrium 32, the left ventricle 33, the leftatrium 34, the interventricular septum 35, and the crux cordis 36correspond to the elements h. The examination image 24 renders ribs 25,the spine 26, the descending aorta 27, the pulmonary artery 37, theascending aorta 38, and the superior vena cava 39. In the presentembodiment, with respect to the time period (the time set T_R) duringwhich the heart is rendered in the examination images after the scan isstarted, probabilities are calculated and displayed with respect to eachof the sites, by comparing the object data R with the site detectiondata D. The examination images 22 to 24 in FIG. 7 are illustrated sothat the sites are easy to see for explanation purposes; however, inactual examination images, the sites would not be clearly displayed inthis manner.

Returning to the description of FIG. 1, the judging unit 132 judgeswhether or not the structure H (the heart) of the object serving as theexamined subject is rendered in the examination image m_t, by using therelationship between the preceding frame and the following frame in themoving image. When having received the input of the moving image fromthe obtaining unit 131, the judging unit 132 extracts an examinationimage m_t in one frame from the input moving image. By making use of thefact that the heart has large pulsating movements, the judging unit 132calculates a score (rule_score) corresponding to the pulsation by usingExpression (1) presented below. In other words, the judging unit 132calculates the score corresponding to the pulsation on the basis of adifference between the frames in the moving image.

$\begin{matrix}{({rule\_ score}) = {\sum\limits_{x,y}\left( {{{m\_ t}\left( {x,y} \right)} - {{m\_ t}\left( {x,y} \right)}} \right)^{2}}} & (1)\end{matrix}$

In Expression (1), x denotes the vertical axis of the screen, whereas ydenotes the horizontal axis of the screen. Further, m_t(x,y) denotes apixel at coordinates (x,y) in the examination image (the frame) taken atthe time t. The entirety of Expression (1) expresses calculating, as thescore, a sum of differences between an examination image taken at a timeearlier than the time t by a time period “a” and the examination imagetaken at the time t. The higher the score is, the larger change isexhibited in the examination images during the time period “a”, i.e.,the heart is pulsating. The value t′ denotes t-a, which is a valueobtained by subtracting the predetermined difference “a” from “t”. Thedifference “a” denotes approximately “1 to 20”, for example. In a 40-fpsmoving image, because the unit of time is 1/40 seconds, the difference“a” is in the range “from 1/40 seconds to ½ seconds” approximately.

The judging unit 132 judges whether or not the examination image m_trenders the structure H (the heart) of the object serving as theexamined subject, by judging whether or not the calculated score exceedsa threshold value k_r set in advance. The threshold value k_r may be anarbitrary value and may be a value such as “3.0”, for example. Whenhaving determined that the examination image m_t renders the structure H(the heart) of the object serving as the examined subject, the judgingunit 132 calculates the set R_t made up of the sites supposed to berendered in the examination image m_t taken at the time t, as expressedin Expression (2) presented below. On the contrary, when havingdetermined that the examination image m_t does not render the structureH (the heart) of the object serving as the examined subject, the judgingunit 132 calculates the object data R_t as expressed in Expression (3)presented below (where R_t=an empty set). In Expression (2), the objectdata R_t rendering the structure H of the object is calculated on thebasis of the six elements h; however, it is acceptable to calculate theobject data R_t on the basis of an arbitrary number of elements h.

R_t={right ventricle, right atrium, left ventricle, left atrium,interventricular septum, crux cordis}  (2)

R_t=ø  (3)

Further, when the object data R_t is calculated as expressed inExpression (2), the judging unit 132 adds the time t to the time setT_R. The judging unit 132 stores the calculated object data R_t and thetime set T_R into the object data storage unit 122. Further, the judgingunit 132 outputs the extracted examination image m_t to the detectingunit 133.

Further, when having received the input of an end judging instructionfrom the detecting unit 133, the judging unit 132 judges whether or notthe moving image has finished. In other words, the judging unit 132judges whether or not the extracted examination image m_t was the finalframe of the moving image. When having determined that the moving imagehas not finished, the judging unit 132 advances the time t by 1,extracts the following examination image m_t from the moving image, andrepeats the process of judging whether or not the structure H (theheart) of the object is rendered. On the contrary, when havingdetermined that the moving image has finished, the judging unit 132outputs a generation instruction to the display controlling unit 134.

When having received the input of the examination image m_t from thejudging unit 132, the detecting unit 133 refers to the learning modelstorage unit 123 and performs a process of detecting the elements h (thesites) of the structure H (the heart) of the object on the inputexamination image m_t, by using the learning model. In other words, thedetecting unit 133 calculates site detection data D_t, which is a setD_t made up of the sites (the elements h) rendered in the examinationimage m_t, by using Expression (4) presented below.

D_t={h|P_h(m_t) is equal to or larger than the threshold value k_d}  (4)

In Expression (4), P_h denotes a probability of the position of theelement h that is calculated when the element h is detected by using thelearning model. The threshold value k_d is a threshold value used forjudging the detection of the element h with respect to the probabilityof the position of the element h. In other words, by using Expression(4), the detecting unit 133 calculates a detection result of each of theelements h (the sites) from the examination image m_t, as the sitedetection data D_t. In this situation, the threshold value k_d may be anarbitrary value and may be a value such as “3.0”, for example. Byadjusting the value of the threshold value “k_d”, it is possible to setacuteness of the detection in a range from “an element being completelymissing” to “deviation from normal data”.

Further, by placing a focus on the probability P_h, it is possible toexpress Expression (4) as a probability map D(h,t) by using Expression(5) presented below.

$\begin{matrix}{{D\left( {h,t} \right)} = \left\{ \begin{matrix}0 & \left( {{{P\_ h}({m\_ t})} < {k\_ d}} \right) \\{{P\_ h}({m\_ t})} & \left( {{{P\_ h}({m\_ t})} \geqq {k\_ d}} \right)\end{matrix} \right.} & (5)\end{matrix}$

The probability P_h and the threshold value k_d in Expression (5) arethe same as those in Expression (4). The detecting unit 133 stores thecalculated site detection data D_t into the site detection data storageunit 124 and also outputs an end judging instruction to the judging unit132.

In other words, the detecting unit 133 performs the site detectingprocess that uses the object detection technique on each of theplurality of ultrasound examination images selected by the judging unit132.

When having received the input of the generation instruction from thejudging unit 132, the display controlling unit 134 refers to the objectdata storage unit 122 and obtains the time set T_R. Further, the displaycontrolling unit 134 refers to the site detection data storage unit 124and obtains the site detection data D corresponding to the time set T_R.On the basis of the time set T_R and the site detection data D, thedisplay controlling unit 134 generates a site detection mapcorresponding to all the examination images M. In this situation, in aframe corresponding to such a time t that is not included in the timeset T_R, because there is supposed to be no site of the examinedsubject, the display controlling unit 134 indicates the site as“undetected”. Further, when generating the site detection map, thedisplay controlling unit 134 may refer to the object data R in theobject data storage unit 122.

In other words, on the basis of the time set T_R and the site detectiondata D, the display controlling unit 134 generates, by using Expression(5), the probability map D(h,t), which is a site detection map in whichdetection results of the sites (the elements h) in each of theexamination images m_t are arranged along the scanning direction (thedirection of the times t). The display controlling unit 134 outputs thegenerated site detection map to the display unit 111 and causes thedisplay unit 111 to display the site detection map.

While generating the site detection map, when the data corresponding toa certain site in the site detection data D_t representing the detectionresults is consecutively in the state of being undetected or exhibitinga probability lower than a predetermined value for a time period equalto or longer than a predetermined number of times t, the displaycontrolling unit 134 displays the part corresponding to such a site in ahighlighted manner. In other words, when a certain site remainsundetected in examination images over a certain span, the displaycontrolling unit 134 is capable of presenting that there is a highpossibility of an abnormality, by displaying such a section of the sitedetection map corresponding to the site in the highlighted manner.

Further, when there is an examination image having a detection result inwhich all of the plurality of sites are indicated as undetected, thedisplay controlling unit 134 is capable of presenting that there is aproblem in the image quality of the examination image by displaying, ina highlighted manner, such a section of the site detection map thatcorresponds to the examination image. Further, in accordance with theindication of being undetected and the probabilities of the detectedsites, for example, the display controlling unit 134 is also capable ofpresenting display corresponding to the probabilities, by displaying thedata in such a manner that the higher the probability is, the morenoticeable color for the user is used for the display of the data, forexample. Further, when any one of the detection results is designated inthe site detection map, the display controlling unit 134 displays anexamination image corresponding to the designated detection result.Further, when detection results in which all of the plurality of sitesare indicated as undetected are consecutively observed in the scanningdirection (the direction of the times t), the display controlling unit134 displays information indicating that the scan is defective.

In other words, on the basis of the detection results, the displaycontrolling unit 134 displays the site detection map in which thedetection result of each of the plurality of sites included in theexamined subject is kept in correspondence with the scan. Further, amongone or more of the sites of which the detection result is indicated asundetected and one or more of the sites of which the detection resultexhibits a probability lower than a predetermined value, when at leastone of such sites is consecutively observed in the scan for a timeperiod equal to or longer than a predetermined length of time, thedisplay controlling unit 134 displays, in a highlighted manner, the partof the site detection map corresponding to the site. In this situation,the predetermined length of time may be a time period in the scandirection corresponding to the time t=6 or longer, for example. Further,with respect to a detection result in which all of the plurality ofsites are indicated as undetected, the display controlling unit 134displays, in a highlighted manner, the part of the site detection mapcorresponding to the detection result. Further, the display controllingunit 134 displays the detection results in different modes in accordancewith the indication of being undetected and the probabilities of thedetection results. Further, when any one of the detection results isdesignated in the site detection map, the display controlling unit 134displays an ultrasound examination image corresponding to the designateddetection result. Further, when detection results in which all of theplurality of sites are indicated as undetected are consecutivelyobserved, the display controlling unit 134 displays informationindicating that the scan is defective.

Next, site detection maps exhibiting various patterns will be explained,with reference to FIGS. 8 to 12. FIG. 8 is a drawing illustrating anexample of the site detection map exhibiting a normal pattern. A sitedetection map 40 illustrated in FIG. 8 is the example of the sitedetection map exhibiting the normal pattern. The site detection map 40illustrated in FIG. 8 indicates values of the probability map D(h,t)while the vertical axis expresses the sites serving as the elements h,whereas the horizontal axis expresses the cross-sectional planes (theexamination images) corresponding to the times t. The site detection map40 indicates that “sites A to F” are detected with sufficientprobabilities over a certain number of cross-sectional planes(examination images). As for the probabilities of the sites on thecross-sectional planes, each of the sites detected with a sufficientprobability is indicated with “S”, while each of the sites detected withan insufficient probability is indicated with “I”, and each of theundetected sites is indicated with “U”, similarly to the site detectionmap 13 illustrated in FIG. 2. Further, on “cross-sectional plane 6”,none of the sites is detected. However, on the preceding and thefollowing cross-sectional planes, certain sites are detected.Accordingly, there is a high possibility that the problem lies in theimage quality of the examination image on “cross-sectional plane 6”,rather than an abnormality in the examined subject. For this reason, bydisplaying a box 41 so as to display the column of “cross-sectionalplane 6” in a highlighted manner, for example, the examination assistingapparatus 100 is able to provide assistance for the examination.

FIG. 9 is a drawing illustrating an example of the site detection mapexhibiting an abnormal pattern. A site detection map 42 illustrated inFIG. 9 is the example of the site detection map exhibiting the abnormalpattern. The site detection map 42 illustrated in FIG. 9 indicatesvalues of the probability map D(h,t) while the vertical axis expressesthe sites serving as the elements h, whereas the horizontal axisexpresses the cross-sectional planes (the examination images)corresponding to the times t. The site detection map 42 indicates that“site A” and “sites C to F” are detected with sufficient probabilitiesover a certain number of cross-sectional planes (examination images) butthat “site B” is not detected over a certain number of cross-sectionalplanes (examination images). Accordingly, in the site detection map 42,there is a high possibility that an abnormality relevant to “site B” maybe present. For this reason, the examination assisting apparatus 100 isable to provide assistance for the examination by, for example,displaying a box 43 so as to display the row of “site B” in ahighlighted manner. Further, the examination assisting apparatus 100 maydisplay, in a highlighted manner, a row corresponding to a site on whicha focus is placed during the examination. Further, in the site detectionmap 42, by displaying the box 41 for the “cross-sectional plane 6” so asto display the column of “cross-sectional plane 6” in a highlightedmanner similarly to the site detection map 40, it is possible to provideassistance for the examination.

FIG. 10 is a drawing illustrating another example of the site detectionmap exhibiting an abnormal pattern. A site detection map 44 illustratedin FIG. 10 is the example of the site detection map exhibiting theabnormal pattern that is different from that in the site detection map42. The site detection map 44 illustrated in FIG. 10 indicates values ofthe probability map D(h,t) while the vertical axis expresses the sitesserving as the elements h, whereas the horizontal axis expresses thecross-sectional planes (the examination images) corresponding to thetimes t. The site detection map 44 indicates that “site A” and “sites Cto F” are detected with sufficient probabilities over a certain numberof cross-sectional planes (examination images) but that “site B” isdetected with low probabilities over a certain number of cross-sectionalplanes (examination images). Accordingly, in the site detection map 44,there is a high possibility that an abnormality relevant to “site B” maybe present, because the shape and the texture or the like of “site B”are different from those in normal situations. For this reason, theexamination assisting apparatus 100 is able to provide assistance forthe examination by, for example, displaying a box 45 so as to displaythe row of “site B” in a highlighted manner. Further, in the sitedetection map 44, by displaying the box 41 for “cross-sectional plane 6”so as to display the column of “cross-sectional plane 6” in ahighlighted manner similarly to the site detection map 40, it ispossible to provide assistance for the examination.

FIG. 11 is a drawing illustrating an example of the site detection mapexhibiting an undeterminable pattern. A site detection map 46illustrated in FIG. 11 is the example of the site detection mapexhibiting the undeterminable pattern. The site detection map 46illustrated in FIG. 11 indicates values of the probability map D(h,t)while the vertical axis expresses the sites serving as the elements h,whereas the horizontal axis expresses the cross-sectional planes (theexamination images) corresponding to the times t. The site detection map46 indicates that certain sites are detected on “cross-sectional plane3” and “cross-sectional plane 4” but that no site is detected on theother cross-sectional planes. When the number of cross-sectional planeson which certain sites are detected is significantly small as observedin this example, the examination assisting apparatus 100 judges that thedetection results are undeterminable and, for example, displays, on thedisplay unit 111, information indicating that the scan performed by theprobe 110 is defective.

FIG. 12 is a drawing illustrating another example of the site detectionmap exhibiting an undeterminable pattern. A site detection map 47illustrated in FIG. 12 is the example of the site detection mapexhibiting the undeterminable pattern different from that in the sitedetection map 46. The site detection map 47 illustrated in FIG. 12indicates values of the probability map D(h,t) while the vertical axisexpresses the sites serving as the elements h, whereas the horizontalaxis expresses the cross-sectional planes (the examination images)corresponding to the times t. The site detection map 47 indicates thatcross-sectional planes on which none of the sites is detected areconsecutively observed in “cross-sectional plane 5” and “cross-sectionalplane 6”. In this manner, when cross-sectional planes on which themajority of the sites are not detected successively continue, theexamination assisting apparatus 100 judges that the detection resultsare undeterminable and displays, on the display unit 111, informationindicating that the scan performed by the probe 110 is defective, forexample.

Next, examples of a site detection map, ultrasound examination images,and site detection results from the ultrasound examination images willbe explained by using an example in which a fetal heart is used as anexamined subject, with reference to FIGS. 13 to 18. FIG. 13 is a drawingillustrating an example of the site detection map. A site detection map48 illustrated in FIG. 13 indicates values of the probability map D(h,t)while the vertical axis expresses the elements h, whereas the horizontalaxis expresses the time t. The site detection map 48 indicates values ofthe probability map D(h,t) by using a gray scale, instead of the symbols“S”, “I”, and “U” used in FIGS. 8 to 12. For example, the site detectionmap 48 indicates the detection results at three levels, namelyundetected, detected (with a probability lower than 20%), and detected(with a probability of 20% or higher). Alternatively, it is alsoacceptable to configure the site detection map 48 so as to display“undetected” in gray and display “detected (with a probability lowerthan 20%)” and “detected (with a probability of 20% or higher)” indifferent colors such as white and blue, for example. Next, byextracting cross-sectional planes Nos. 235 to 285 indicated with a box49, ultrasound examination images and site detection results will beexplained.

FIG. 14 is a drawing illustrating examples of ultrasound examinationimages. Among the extracted cross-sectional planes in FIG. 13, FIG. 14illustrates ultrasound examination images corresponding to Nos. 235,245, 255, 265, 275, and 285. In these ultrasound examination images,shadows are observed over the cross-sectional planes corresponding toNos. 245 and 255.

FIG. 15 is a drawing illustrating examples of site detection results.Among the ultrasound examination images in FIG. 14, FIG. 15 illustratessite detection results corresponding to Nos. 235, 255, and 275. In No.235, the spine 26, the descending aorta 27, the right ventricle 31, theright atrium 32, the left ventricle 33, the left atrium 34, theinterventricular septum 35, and the crux cordis 36 are detected. In No.255, the spine 26, the descending aorta 27, the right ventricle 31, theright atrium 32, and the interventricular septum 35 are detected. Inother words, in No. 255, because of the shadows, the left ventricle 33,the left atrium 34, and the crux cordis 36 are not detected. In No. 275,the spine 26, the descending aorta 27, the right ventricle 31, the rightatrium 32, the left ventricle 33, the left atrium 34, theinterventricular septum 35, and the crux cordis 36 are detected.

Accordingly, among the cross-sectional planes corresponding to Nos. 235to 285 that represent the sections extracted from the site detection map48, the undetected sites in No. 255 are detected on the cross-sectionalplanes preceding and following No. 255, it is understood that the sitesare normal. In other words, it is understood that the shadows oncross-sectional planes No. 245 and 255 are caused by the scan performedby the probe 110. Further, the examination assisting apparatus 100 maydisplay, in a real-time manner, boxes indicating detection results ofthe sites as illustrated in FIG. 15, within the ultrasound examinationimages for the monitor that are displayed on the display unit 111 duringthe scan performed by the probe 110. Further, the examination assistingapparatus 100 may display labels indicating the names of the sites orthe like, for example, with the boxes indicating the detection resultsof the sites illustrated in FIG. 15.

FIG. 16 is a drawing illustrating another example of the site detectionmap. Similarly to the site detection map 48 illustrated in FIG. 13, asite detection map 50 illustrated in FIG. 16 indicates values of theprobability map D(h,t) while the vertical axis expresses the elements h,whereas the horizontal axis expresses the time t. The site detection map50 indicates detection results with tetralogy of Fallot, which is atypical congenital heart disease. From the site detection map 50, it isobserved that the crux cordis 36 is missing (a hole in theinterventricular septum 35) and the pulmonary artery 37 is missing,which reflect symptoms of tetralogy of Fallot. Examples of thecross-sectional planes corresponding to the missing crux cordis 36include No. 131. Examples of the cross-sectional planes corresponding tothe mixing pulmonary artery 37 include no. 212.

FIG. 17 is a drawing illustrating other examples of ultrasoundexamination images. FIG. 17 illustrates the ultrasound examinationimages corresponding to Nos. 131 and 212 in FIG. 16. From No. 131, it isobserved that the crux cordis 36 is missing. From No. 212, it isobserved that the pulmonary artery 37 is narrowed due to a stenosis. Asfor relationships between abnormal sites and diseases, for example, whenthe crux cordis 36 is an abnormal site, corresponding diseases are acomplete/incomplete atrioventricular septal defect and tetralogy ofFallot (TOF). As another example, when the crux cordis 36 and theinterventricular septum 35 are abnormal sites, a corresponding diseaseis a single left ventricle defect. As yet another example, when the leftatrium 34 is an abnormal site, corresponding diseases are pulmonaryatresia with intact ventricular septum and Ebstain anomaly. As yetanother example, when the pulmonary artery 37 is an abnormal site, acorresponding disease is mitral atresia. As yet another example, whenthe descending aorta 27 and the spine 26 are abnormal sites,corresponding diseases are transposition of the great arteries, truncusarteriosus, and tetralogy of Fallot (TOF).

FIG. 18 is a drawing illustrating other examples of the site detectionresults. FIG. 18 illustrates the site detection results regarding Nos.131 and 212 in FIG. 17. In No. 131, the descending aorta 27, the rightventricle 31, the right atrium 32, the left ventricle 33, the leftatrium 34, the interventricular septum 35, and the ascending aorta 38are detected. In contrast, in No. 131, the crux cordis 36 is missing andtherefore not detected. In No. 212, the descending aorta 27 and theascending aorta 38 are detected. In contrast, in No. 212, the pulmonaryartery 37 is narrowed due to the stenosis and is therefore not detected.In other words, from the site detection map 50, in correspondence withthe observation that the crux cordis 36 and the pulmonary artery 37 aremissing, it is also possible to recognize, in the correspondingultrasound examination images, that those sites are missing.

Next, an operation of the examination assisting apparatus 100 accordingto the first embodiment will be explained. FIG. 19 is a flowchartillustrating an example of an examination assisting process according tothe first embodiment.

When being instructed to start obtaining a moving image, the obtainingunit 131 starts obtaining a moving image on the basis of the receptiondata obtained from the probe 110 (step S1). The obtaining unit 131stores the obtained moving image into the image storage unit 121 andalso outputs the obtained moving image to the judging unit 132.

When having received the input of the moving image from the obtainingunit 131, the judging unit 132 extracts an examination image m_t in oneframe from the input moving image (step S2). The judging unit 132calculates object data R_t from the extracted examination image m_t(step S3). When having determined that the examination image m_t rendersthe structure H (the heart) of the object serving as the examinedsubject, the judging unit 132 adds the time t to the time set T_R (stepS4). The judging unit 132 stores the calculated object data R_t and thetime set T_R into the object data storage unit 122. Further, the judgingunit 132 outputs the extracted examination image m_t to the detectingunit 133.

When having received the input of the examination image m_t from thejudging unit 132, the detecting unit 133 refers to the learning modelstorage unit 123 and performs the process of detecting the elements h(the sites) of the structure H (the heart) of the object on the inputexamination image m_t, by using the learning model. In other words, thedetecting unit 133 calculates site detection data D_t from theexamination image m_t (step S5) The detecting unit 133 stores thecalculated site detection data D_t into the site detection data storageunit 124 and also outputs an end judging instruction to the judging unit132.

When having received the input of the end judging instruction from thedetecting unit 133, the judging unit 132 judges whether or not themoving image has finished (step S6). When having determined that themoving image has not finished (step S6: No), the judging unit 132advances the time t by 1, i.e., advances the moving image by one frame,and returns to step S2. On the contrary, when having determined that themoving image has finished (step S6: Yes), the judging unit 132 outputs ageneration instruction to the display controlling unit 134.

When having received the input of the generation instruction from thejudging unit 132, the display controlling unit 134 refers to the objectdata storage unit 122 and obtains the time set T_R. Further, the displaycontrolling unit 134 refers to the site detection data storage unit 124and obtains the site detection data D corresponding to the time set T_R.On the basis of the time set T_R and the site detection data D, thedisplay controlling unit 134 generates a site detection map (step S7).The display controlling unit 134 outputs the generated site detectionmap to the display unit 111 and causes the display unit 111 to displaythe site detection map (step S8). As a result, the examination assistingapparatus 100 is able to provide the site detection map that makes itpossible to easily judge whether an abnormality is present or absent.Further, even when the level of precision of the scan is not sufficientor when the examined subject significantly changes in the course oftime, the examination assisting apparatus 100 is able to provideinformation useful for diagnosing processes by displaying the detectionresults of the sites (the results of the object recognition)chronologically. Further, because using the site detection map enablesthe viewer to see, in a list view, the manner in which the sites aredetected in the entire moving image of the examined subject, theexamination assisting apparatus 100 is able to reduce the time involvedin the viewer's checking process. Further, because the examinationassisting apparatus 100 makes it possible to identify the sitesregardless of the skills of users (examiners), it is possible to keepvariations in examination results small among the users.

When the present embodiment is compared with related ultrasoundexaminations, during a related ultrasound examination, an examinationmay be performed depending on whether a specific site is properlyrendered in an image of a two-dimensionally scanned cross-sectionalplane. The quality of the image of the two-dimensionally scannedcross-sectional plane from the examination is not stable due to noise,shadows, movements of the examined subject, and the like. It wouldtherefore be difficult to directly detect the sites. In contrast, theexamination assisting apparatus 100 according to the present embodimentperforms the site detecting process that uses the object detectiontechnique on each of the plurality of ultrasound examination images anddisplays the site detection map in which the detection result of each ofthe plurality of sites included in the examined subject is kept incorrespondence with the scan. As a result, the examiner is able toeasily perform the examination by referring to the site detection map.

Next, an ultrasound examination using the examination assistingapparatus 100 according to the present embodiment will be compared witha related ultrasound examination. For example, during the relatedultrasound examination, an examination may be performed by checking fora plurality of sites in an image of a two-dimensionally scannedcross-sectional plane obtained by using an ultrasound wave. As for theimage of the two-dimensionally scanned cross-sectional plane obtained bythe ultrasound examination, what is rendered in the image and thequality of the image are not stable due to a plurality of factorsincluding noise, a shadow of a human arm or the like, movements of theexamined subject such as pulsation of the heart, and the like. In otherwords, from among the plurality of images taken on the two-dimensionallyscanned cross-sectional planes, the examiner has to check each of aplurality of sites from one or more of the images in which the pluralityof sites are recognizable. For this reason, the related ultrasoundexamination involves skillful and experienced work to identifyappropriate images from which each of the plurality of sites is to bechecked and to check the statuses of the sites in the identified images.Accordingly, obtaining an appropriate examination result is dependent onthe skills of the examiner.

In contrast, the examination assisting apparatus 100 according to thepresent embodiment performs the site detecting process that uses theobject detection technique on each of the plurality of ultrasoundexamination images and displays the site detection map in which thedetection result of each of the plurality of sites included in theexamined subject is kept in correspondence with the scan. By performingan ultrasound examination while using this function, the examiner whorefers to the site detection map does not have to identify the images inwhich the plurality of sites are recognizable. For this reason, evenexaminers who are not experienced and skillful are able to performexaminations with stable quality. Further, when a third person checksexamination data in a retrospective manner, even when the examinationdata is a moving image of two-dimensionally scanned cross-sectionalplanes obtained from an ultrasound examination, he/she who refers to thesite detection map is able to check the examination without having toplay back or check the moving image.

In this manner, on each of the plurality of ultrasound examinationimages taken of the examined subject by performing the scan thereon, theexamination assisting apparatus 100 performs the site detecting processthat uses the object detection technique. Further, on the basis of thedetection results, the examination assisting apparatus 100 displays thesite detection map in which the detection result of each of theplurality of sites included in the examined subject is kept incorrespondence with the scan. As a result, the examination assistingapparatus 100 is able to provide the site detection map that makes itpossible to easily judge whether an abnormality is present or absent.

Further, among one or more of the sites of which the detection result isindicated as undetected and one or more of the sites of which thedetection result exhibits a probability lower than a predeterminedvalue, when at least one of such sites is consecutively observed in thescan for a time period equal to or longer than the predetermined lengthof time, the examination assisting apparatus 100 displays, in ahighlighted manner, the part of the site detection map corresponding tothe site. As a result, the examination assisting apparatus 100 is ableto advise about the site having a high possibility of being abnormal.

Further, with respect to a detection result in which all of theplurality of sites are indicated as undetected, the examinationassisting apparatus 100 displays, in a highlighted manner, the part ofthe site detection map corresponding to the detection result. As aresult, the examination assisting apparatus 100 is able to advise aboutthe detection result (the examination image) having a high possibilityof the scan being defective.

Further, the examination assisting apparatus 100 displays the detectionresults in the different modes in accordance with the indication asbeing undetected and the probabilities of the detection results. As aresult, the examination assisting apparatus 100 is able to provide thesite detection map having a high degree of at-a-glance convenience.

Further, when any one of the detection results is designated in the sitedetection map, the examination assisting apparatus 100 displays theultrasound examination image corresponding to the designated detectionresult. As a result, the examination assisting apparatus 100 is able todisplay the ultrasound examination image corresponding to the detectionresult having a high possibility of being abnormal in the site detectionmap.

Further, when detection results in which all of the plurality of sitesare indicated as undetected are consecutively observed, the examinationassisting apparatus 100 displays the information indicating that thescan is defective. As a result, the examination assisting apparatus 100is able to prompt the user to perform a re-examination.

Further, the scan performed by the examination assisting apparatus 100is a scan in a forward direction. As a result, the examination assistingapparatus 100 is able to display the site detection map corresponding tothe scan performed in the forward direction.

Further, the scan performed by the examination assisting apparatus 100is one selected from among: a slide scan, a rotation scan, a fan-shapedscan, a pendulum scan, and a scan combining any of these scans. As aresult, the examination assisting apparatus 100 is able to display thesite detection map corresponding to the scanning method being used.

Next, a related technique used for presenting information about anarbitrary cross-sectional plane based on a scan result obtained byimplementing CT, MRI, or the like will be compared with the technique ofthe present embodiment used for generating and displaying the sitedetection map.

When the information about the cross-sectional plane is presented on thebasis of the scan result obtained by implementing CT, MRI, or the like,generating a three-dimensional model involves measured data having asufficient level of precision. Further, such a technique expects thatthe measured data does not significantly vary in the course of time.However, it is difficult to obtain such measured data that has asufficient level of precision from ultrasound examinations. It wouldtherefore be difficult to structure a three-dimensional model to presentthe information about the cross-sectional plane by using the relatedtechnique.

In contrast, the examination assisting apparatus 100 according to thepresent embodiment is able to perform the site detecting process thatuses the object detection technique on each of the plurality ofultrasound examination images and to display the site detection map inwhich the detection result of each of the plurality of sites included inthe examined subject is kept in correspondence with the scan.

[b] Second Embodiment

In the first embodiment above, the example using the fetal heart as theexamined subject is explained. However, the present disclosure isapplicable to any object as long as it is possible to perform anultrasound examination thereon. For example, the examined subject may bea semiconductor package. An embodiment in this situation will beexplained as a second embodiment. In the second embodiment, only theexamined subject is different from that of the examination assistingapparatus 100 according to the first embodiment. Thus, the examinedsubject will be explained in the following sections, and duplicateexplanations of configurations and operations will be omitted.

In recent years, in more and more semiconductor packages, a large numberof chips and the like (components) are installed in a single packagecalled a System in Package (SiP), owing to the advancement ofthree-dimensional mounting technology. Because such semiconductorpackages have a complicated internal structure, there is a demand notonly for the capability of detecting flaws by performing a relatedultrasound examination, but also for the capability of checking theinstallation status of the internal structure.

FIG. 20 is a drawing illustrating examples of an examined subject and anexamination method according to the second embodiment. A package 60illustrated in FIG. 20 is an example of a semiconductor package. In thepackage 60, a plurality of chips 62 are mounted on a substrate 61 andare sealed by package resin 63. In the second embodiment, the package 60is placed at the bottom of a water tank 64, so as to obtain anexamination image by moving a probe 65 parallel to the surface of thewater in the water tank 64. The probe 65 corresponds to the probe 110 ofthe examination assisting apparatus 100.

In the second embodiment, the examination assisting apparatus 100calculates object data R_t on the basis of a design drawing of thepackage 60. In other words, in the second embodiment, on the basis ofthe design drawing of the package 60, the examination assistingapparatus 100 calculates the object data R_t by obtaining the types ofcomponents that are supposed to be rendered in the examination image m_trepresenting a cross-sectional plane and the time t. Further, incorrespondence with the time periods in which the components aresupposed to be rendered, the examination assisting apparatus 100calculates a time set T_R. In this situation, because the calculation ofthe object data R_t is performed in the same manner as in the firstembodiment, the explanation thereof will be omitted.

FIG. 21 is a drawing illustrating examples of the components that aresupposed to be rendered. As illustrated in FIG. 21, a plan view 66 ofthe package 60 indicates a plurality of chips 62 a, 62 b, and 62 c asthe components that are supposed to be rendered. In the secondembodiment, when a cross-sectional plane 67 is moved in a probe scanningdirection, a site detection map, i.e., a probability map D(h,t) isgenerated with respect to the chips 62 a, 62 b, and 62 c. In the secondembodiment, the time set T_R is made up of times t corresponding to thetime period from the point at which the chip 62 a appears on thecross-sectional plane 67 to the point at which the chip 62 b disappearsfrom the cross-sectional plane 67.

The examination assisting apparatus 100 calculates site detection dataD_t from the examination images m_t. On the basis of the time set T_Rand the site detection data D corresponding to the time set T_R, theexamination assisting apparatus 100 generates the site detection map,i.e., the probability map D(h,t). The display controlling unit 134outputs the generated site detection map to the display unit 111 andcauses the display unit 111 to display the site detection map. Becausethe calculation of the site detection data D_t is performed in the samemanner as in the first embodiment, the explanation thereof will beomitted.

FIG. 22 is a drawing illustrating an example of the site detection mapaccording to the second embodiment. A site detection map 68 illustratedin FIG. 22 is an example of the site detection map corresponding to theplan view 66 of the package 60. The site detection map 68 illustrated inFIG. 22 indicates values of the probability map D(h,t) while thevertical axis expresses the chips serving as the elements h, whereas thehorizontal axis expresses the cross-sectional planes (the examinationimages) corresponding to the times t. As illustrated in FIG. 22, thesite detection map 68 displays the probability of each of thecross-sectional planes with respect to each of the chips 62 a, 62 b, and62 c. In this situation, for example, when the package resin 63 has anair bubble, although it is not possible to see a chip 62 positionedunderneath the air bubble by simply looking at the examination image ofthe cross-sectional plane, it is possible to learn that the chip 62 ispresent by looking at the probabilities of the cross-sectional planes.Further, from the site detection map 68, it is possible to recognize, ata glance, the installation state including a relationship among thepositions of the chips 62 a, 62 b, and 62 c. As explained herein, evenwhen the examined subject is the semiconductor package, the examinationassisting apparatus 100 according to the second embodiment is capable ofproviding the site detection map on the basis of the results from theultrasound examination images.

In the embodiments described above, the scanning direction is indicatedin FIGS. 13, 16, 20, and 21; however, possible scanning directions arenot limited to those illustrated in the drawings. For instance, it isalso acceptable to identify the scanning direction by comparing theframes in the moving image with one another, to calculate the objectdata R_t and the site detection data D_t from a set made up ofexamination images in a predetermined span excluding duplicate frames,and to generate a site detection map of the predetermined span.

In the embodiments described above, YOLO, SSD, and Faster-RNN arementioned as examples of the object detection algorithms; however,possible examples are not limited to these. For instance, it isacceptable to use an object detection algorithm using any of varioustypes of neural networks such as DPM, Fast-RNN, or the like. Further, asthe learning method, it is possible to use any of various publicly-knownmethods other than backpropagation methods. Further, neural networkshave, for example, a multilayer structure including an input layer, anintermediate layer (a hidden layer), and an output layer, while thelayers have such a structure in which a plurality of nodes are connectedby edges. Each of the layers has a mathematical function called an“activation function”. Each of the edges has a “weight”. The value ofeach of the nodes is calculated from the values of the nodes in aprevious layer, the values of the weights of the connecting edges, andthe activation function of the layer. As for the calculation method, itis possible to use any of various publicly-known methods. As for themachine learning, besides the neural network scheme, it is possible touse any of various methods such as a Support Vector Machine (SVM)method.

The constituent elements of the functional units illustrated in thedrawings do not necessarily have to be physically configured asindicated in the drawings. In other words, specific modes ofdistribution and integration of the functional units are not limited tothose illustrated in the drawings. It is acceptable to functionally orphysically distribute or integrate all or a part of the functional unitsin any arbitrary units, depending on various loads, the status of use,and the like. For example, the judging unit 132 and the detecting unit133 may be integrated together. Further, the processes illustrated inthe drawings do not have to be performed in the order described above.As long as no conflict occurs in the processing, it is acceptable toperform certain processes simultaneously and to change the order inwhich certain processes are performed.

Further, as for the various types of processing functions implemented bythe apparatuses and the devices, all or an arbitrary part thereof may beexecuted by a Central Processing Unit (CPU) (or a microcomputer such asa Micro Processing Unit (MPU), a Micro Controller Unit (MCU), or thelike). Further, needless to say, it is acceptable to execute all or anarbitrary part of the various types of processing functions by using aprogram analyzed and executed by a CPU (or a microcomputer such as anMPU, an MCU, or the like) or by using hardware structured with wiredlogic.

Further, it is possible to realize the various types of processesexplained in the above embodiments by causing a computer to execute aprogram prepared in advance. Thus, in the following sections, an exampleof the computer that executes a program having the same functions asthose described in the above embodiments will be explained. FIG. 23 is adrawing illustrating an example of a computer that executes anexamination assisting program.

As illustrated in FIG. 23, a computer 200 includes a CPU 201 thatexecutes various type of computing processes, an input device 202 thatreceives data inputs, and a monitor 203. Further, the computer 200includes a medium reading device 204 that reads a program and the likefrom a storage medium, an interface device 205 used for connecting tovarious types of devices, and a communication device 206 used forconnecting in a wired or wireless manner to other information processingapparatuses or the like. Further, the computer 200 includes a RAM 207that temporarily stores therein various types of information and a harddisk device 208. Further, the devices 201 to 208 are connected to a bus209.

The hard disk device 208 stores therein the examination assistingprogram having the same functions as those of processing unitsillustrated in FIG. 1 such as the obtaining unit 131, the judging unit132, the detecting unit 133, and the display controlling unit 134.Further, the hard disk device 208 stores therein various types of datafor realizing the image storage unit 121, the object data storage unit122, the learning model storage unit 123, the site detection datastorage unit 124, and the examination assisting program. For example,the input device 202 receives inputs of various types of informationsuch as operation information, from a user of the computer 200. Forexample, the monitor 203 displays various types of screens such asdisplay screens, for the user of the computer 200. For example, theinterface device 205 has a probe or the like connected thereto. Forexample, the communication device 206 is connected to a network (notillustrated) and exchanges various types of information with the otherinformation processing apparatuses.

The CPU 201 performs various types of processes by reading programsstored in the hard disk device 208, loading the read programs into theRAM 207, and executing the programs. Further, these programs are able tocause the computer 200 to function as the obtaining unit 131, thejudging unit 132, the detecting unit 133, and the display controllingunit 134 illustrated in FIG. 1.

The examination assisting program described above does not necessarilyhave to be stored in the hard disk device 208. For example, it is alsoacceptable to configure the computer 200 to read and execute a programstored in a storage medium that is readable by the computer 200.Examples of the storage medium that is readable by the computer 200include portable recording media such as a Compact Disk Read-Only Memory(CD-ROM), a Digital Versatile Disc (DVD), and a Universal Serial Bus(USB) memory; semiconductor memory elements such as a flash memory; ahard disk drive, and the like. Further, it is also acceptable to storethe examination assisting program in apparatuses connected to a publiccommunication line, the Internet, a Local Area Network (LAN), and thelike so that the computer 200 reads and executes the examinationassisting program from any of these apparatuses.

It is possible to provide the site detection map that makes it possibleto easily judge whether an abnormality is present or absent.

All examples and conditional language recited herein are intended forpedagogical purposes of aiding the reader in understanding the inventionand the concepts contributed by the inventors to further the art, andare not to be construed as limitations to such specifically recitedexamples and conditions, nor does the organization of such examples inthe specification relate to a showing of the superiority and inferiorityof the invention. Although the embodiments of the present invention havebeen described in detail, it should be understood that the variouschanges, substitutions, and alterations could be made hereto withoutdeparting from the spirit and scope of the invention.

What is claimed is:
 1. A non-transitory computer-readable recordingmedium storing therein an examination assisting program that causes acomputer to execute a process comprising: performing a site detectingprocess that uses an object detection technique on each of a pluralityof ultrasound examination images taken of an examined subject byperforming a scan on the examined subject; and displaying a sitedetection map in which a detection result of each of a plurality ofsites included in the examined subject is kept in correspondence withthe scan, on a basis of detection results from the site detectingprocess.
 2. The non-transitory computer-readable recording mediumaccording to claim 1, wherein, among one or more of the sites of whichthe detection result is indicated as undetected and one or more of thesites of which the detection result exhibits a probability lower than apredetermined value, when at least one of the sites is consecutivelyobserved in the scan for a time period equal to or longer than apredetermined length of time, the displaying includes displaying, in ahighlighted manner, a part of the site detection map corresponding tothe site.
 3. The non-transitory computer-readable recording mediumaccording to claim 1, wherein, with respect to any of the detectionresults in which all of the plurality of sites are indicated asundetected, the displaying includes displaying, in a highlighted manner,a part of the site detection map corresponding to the detection result.4. The non-transitory computer-readable recording medium according toclaim 1, wherein the displaying includes displaying the detectionresults in different modes in accordance with indication of beingundetected and probabilities of the detection results.
 5. Thenon-transitory computer-readable recording medium according to claim 1,wherein, when any one of the detection results is designated in the sitedetection map, the displaying includes displaying one of the ultrasoundexamination images corresponding to the designated detection result. 6.The non-transitory computer-readable recording medium according to claim1, wherein, when detection results in which all of the plurality ofsites are indicated as undetected are consecutively observed, thedisplaying includes displaying information indicating that the scan isdefective.
 7. The non-transitory computer-readable recording mediumaccording to claim 1, wherein the scan is a scan performed in a forwarddirection.
 8. The non-transitory computer-readable recording mediumaccording to claim 1, wherein the scan is one selected from among: aslide scan, a rotation scan, a fan-shaped scan, a pendulum scan, and acombination of any of these scans.
 9. An examination assisting methodimplemented by a computer to perform processes of: performing a sitedetecting process that uses an object detection technique on each of aplurality of ultrasound examination images taken of an examined subjectby performing a scan on the examined subject, by a processor; anddisplaying a site detection map in which a detection result of each of aplurality of sites included in the examined subject is kept incorrespondence with the scan, on a basis of detection results from thesite detecting process.
 10. An examination assisting apparatuscomprising: a processor configured to: perform a site detecting processthat uses an object detection technique on each of a plurality ofultrasound examination images taken of an examined subject by performinga scan on the examined subject; and display a site detection map inwhich a detection result of each of a plurality of sites included in theexamined subject is kept in correspondence with the scan, on a basis ofdetection results from the site detecting process.